1. Technical Field
The present invention relates generally to measuring the performance of streaming media delivery.
2. Description of the Related Art
Streaming media is a type of Internet content that has the important characteristic of being able to be played while still in the process of being downloaded. A client machine can play the first packet of the stream, and decompress the second, while receiving the third. Thus, an end user can start enjoying the multimedia without waiting to the end of transmission. Streaming is very useful for delivering media because media files tend to be large, particularly as the duration of the programming increases. Indeed, for live events, the file size, in effect, is infinite. To view a media file that is not streamed, users must first download the file to a local hard disk—which may take minutes or even hours—and then open the file with player software that is compatible with the file format. To view streaming media, the user's browser opens player software, which buffers the file for a few seconds and then plays the file while simultaneously downloading it. Unlike software downloads, streaming media files are not stored locally on a user's hard disk. Once the bits representing content are used, the player typically discards them.
Streaming media usually relies on proprietary server and client software. The server, client, production and encoding tools developed by a streaming software vendor are collectively referred to as a format. Streaming media encoded in a particular format must be served by that format's media server and replayed by that format's client. Streaming media clients are often called players, and typically they exist as plug-ins to Web browsers. The three major streaming media formats in use today are: RealNetworks RealSystem G2, Microsoft Windows Media Technologies (“WMT”), and Apple QuickTime.
An ever-growing number of content providers, enterprises, and content delivery networks utilize and deliver streaming media. As a consequence, it has become important to quantify, in a scientific and measurable way, how well a given stream was delivered, as well as to devise relevant metrics that capture the end-user's experience of the stream. Client media players have the capability of generating basic statistics about the streams they deliver. Unfortunately, the correlation of some of these statistics with a high quality end-user experience is subjective and thus extremely difficult to measure effectively. For example, frame rate, a commonly used metric reported by media players, may not be a robust indicator of stream quality because players can interpolate and insert variable-quality frames into a stream. As another example, some media players report a bandwidth metric that does not accurately indicate whether or not the bytes are received in a timely fashion so that they can be used for rendering.
There are several companies, such as Keynote, StreamCheck, and others, that provide partially or completely-outsourced streaming monitoring solutions. In an illustrative system, streaming measurement computers (sometimes called “agents”) connect to URLs and collect relevant statistics, such as initial buffer time, packet information, and overall stream quality. These measurement agents are typically positioned strategically around the Internet so as to accurately represent the experience of an end user connecting to a given site URL. Providers of such outsourced performance monitoring solutions typically provide performance data in the form of real-time and historical reports, and threshold-based alarms.
By way of additional background, it is also known in the prior art that streaming performance measurement is significantly harder than web download measurements for several reasons. Unlike web performance, where it suffices to measure object download times, streams can show degradation in significantly complex ways that need to be quantified and measured. Unlike the HTTP protocol, an open protocol, measuring stream performance involves taking into account complex proprietary streaming protocols, such as RTSP (Real) and MMS (Windows Media).
One can distinguish three steps in making streaming media content available to the end-user: the media is captured and encoded, the media is delivered over the network to the client, i.e., the media player of the end-user, and the media is rendered by the media player to be viewed by the end-user. The process of delivering streaming media over the network can degrade the quality perceived by the end-user. A primary goal of a content delivery network (or “CDN”) is to ensure that the media is delivered to a requesting end-user with as few distortions as possible from its pristine encoded form, thereby maximizing the end-user experience of the stream. The root causes of degradation in stream quality are problems that may occur in the underlying network that transports the media from the server to the client. For example, network connectivity may go down, which limits the ability of the client to talk to the server. The effective bandwidth available between the server and client may be greatly reduced. The network may lose packets sent from the server to the client. More subtly, the network may cause packets to be re-ordered in route to the client. Or, the packets may be delivered by the network to the client late, in which case the packets will not be useful in the stream playback.
It is instructive to examine how the client and the server react to these network problems in a manner that degrades stream quality. As noted above, typically the client and the server communicate using a proprietary protocol (such as Real's RTSP or Windows Media's MMS) to control the manner in which data is streamed from the server to the client. The client-server protocol is typically designed to overcome moderate amounts of network degradation. The player typically buffers several seconds of data ahead of what is currently being rendered. This buffering helps the player smooth out any variations in the rate at which packets are being received, including out-of-order delivery of packets. Further, buffering provides an opportunity for the player to tolerate a certain amount of packet loss, because lost packets can be re-requested from the server and received in time for playback. Consequently, a moderate degradation in the network is effectively shielded from the user.
As the network degrades further, however, one might see prolonged periods of reduction in the available bandwidth, or a significant amount of packets lost, or a significant number of packets arriving late. The server typically reacts to such a situation by sending less than the full encoded bandwidth to the client, resulting in a thinned playback. The server achieves this thinning by sending only the “more important” bytes and throwing away the rest. For example, the server may send only key video frames and audio and not send the rest of the content. In the extreme case, thinning results in the so-called “slide-show” effect, where isolated frames are displayed without interpolation. If the stream is encoded at multiple bit rates, thinning is effected more seamlessly, e.g., by the server switching to a replica of the stream encoded at a lower bit rate. Typically, the bandwidth streamed from the server to the client is controlled by a feedback loop, where the client continually estimates the available bandwidth and reports it to the server, and the server attempts to fit the stream to the available bandwidth. An even more drastic degradation in the network, e.g., a rapid reduction in the available bandwidth, may deplete the player's buffer of data. When the player cannot continue the playback, it freezes to refill its buffer before resuming again, resulting in an interrupted playback. The player is typically designed to rebuffer only under extreme network conditions.
There is a need in the art to provide for improved methods for measuring stream quality and performance. The present invention addresses this need.